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Abstract 

For any family of measurable sets in a probability space, we show that either (i) the 
family has infinite Vapnik-Chervonenkis (VC) dimension or (ii) for every e > there is 
a finite partition ir such the 7r-boundary of each set has measure at most e. Immediate 
corollaries include the fact that a family with finite VC dimension has finite bracketing 
numbers, and satisfies uniform laws of large numbers for every ergodic process. From 
these corollaries, we derive analogous results for VC major and VC graph families of 
functions. 
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1 Introduction 



Let (X,S,fj,) be a probability space and let C C 5 be a given family of measurable sets. 
The Vapnik-Chervonenkis dimension of C is a measure of its combinatorial complexity, 
specifically, the ability of C to separate finite sets of points. Given a finite set D C X, let 
{C (~l -D : C G C} be the collection of subsets of D selected by the members of C. The 
family C is said to shatter D if its elements can select every subset of D, or equivalently, if 
\{C D D : C G C}\ = 2' D i. Here and in what follows, \A\ denotes the cardinality of a given 
set A. The Vapnik-Chervonenkis (VC) dimension [TT] of C, denoted dim(C), is the largest 
integer k such that C is able to shatter some set of cardinality k. If C can shatter arbitrarily 
large finite sets, then dim(C) = +00. A family of sets C is said to be a VC class if dim(C) 
is finite. 

Let 7T be a finite, measurable partition of X. For every set C G C, the 7r-boundary of C, 
denoted d(C : tt), is the union of all the cells in tt that intersect both C and its complement 
with positive probability. Formally, 

d(C : tt) = U {A G tt : n C) • n C c ) > 0}. 

Note that d(C : tt) depends on /i, though this dependence is suppressed in our notation. We 
will call a family C finitely approximable if for every e > there exists a finite, measurable 
partition tt of X such that n{d{C : tt)) < e for every C G C. Our principal result is the 
following. 

Theorem 1. Let (X,S,n) be a probability space and let C Q S be any family of sets. Then 
either (i) C is finitely approximable or (ii) C has infinite VC dimension. 

Theorem [1] extends immediately to finite positive measures; we restrict attention to the 
case of probability measures for simplicity. Gaenssler and Stute [8] studied 7r-boundaries 
in work on uniform convergence of measures. In conjunction with Theorem [TJ their results 
show that, if for some VC-class C and some sequence {/i n } of finite measures, fJ, n (A) — > fJ,(A) 
for every A G cr(C), then this convergence is uniform over C. One may establish the same 
conclusion using Corollary [TJ 

In general, alternatives (i) and (ii) of Theorem [TJ are not mutually exclusive: there exist 
families C that are finitely approximable and have infinite VC dimension. Moreover the finite 
approximability of C will generally depend on the measure [i. To take a simple example, 
let C be the family of all Borel measurable subsets of the unit interval [0, 1]. Then C clearly 
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has infinite VC dimension. An easy argument shows that C is finitely approximable if /x 
has countable support, but that C is not finitely approximable if [i is absolutely continuous 
with respect to Lebesgue measure. As the following, equivalent, version of Theorem Q] 
makes clear, families with finite VC dimension are finitely approximable for any probability 
measure [i. 

Theorem 2. Let (X, S) be a measurable space. If C C S has finite VC dimension, then C 
is finitely approximable for any probability measure fj,. 

Families of sets with finite VC-dimension figure prominently in machine learning, em- 
pirical process theory and combinatorial geometry (c./. [11], [15l [6l El [XHl HO]) and have been 
widely studied in these fields. The majority of this work concerns the combinatorial proper- 
ties of VC-classes, and related exponential probability inequalities for uniform laws of large 
numbers under independent sampling (see Section [3] below). The uniform approximation 
guaranteed by Theorem [2] provides new insights into the structure of VC-classes. 

Some immediate corollaries of Theorem [2] are explored in Sections [2] and [3] below, in- 
cluding new results on the bracketing properties of VC major and VC graph classes of 
functions. Approximation properties analogous to those of Theorems [T] and [2] may be es- 
tablished for classes of functions with finite fat-shattering (gap) dimension [9] by extending 
the arguments in Section HI 

The proof of Theorem Q] makes use of an equivalent version of the VC dimension that 
we now describe. Recall that the join of k sets A\,. . . ,Ak C X, denoted J = Vi=i 
is the finite partition of X consisting of all non-empty intersections A\ fl • • • fl A^, where 
Ai £ {Ai, Af} for i = 1, . . . , k. Equivalently, J consists of the non-empty atoms of the field 
generated by A\, . . . , A^. The collection A%, . . . , A^ C X is said to be Boolean independent 
if J has (maximal) cardinality 2 k . The dual VC dimension, denoted dim*(C), is the largest 
k such that C contains k Boolean independent sets. If C contains Boolean independent 
families of every finite size, then dim*(C) = +oo. The dual VC-dimension was introduced 
by Assouad [I], and is so named because dim*(C) is the VC-dimension of the dual family 
{D x : x G X} C 2 C , where D x = {C € C : x € C}. We will make use of the following, 
elementary result, whose proof can be found in [3], see also [TOl [JJ. 

Lemma A. Let C be any collection of subsets of X . The VC-dimension dim(C) is finite if 
and only if the dual VC-dimension dim*(C) is finite. 

In proving Theorem [1] we begin with the assumption that C is not finitely approximable, 
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and then deduce from this that dim*(C) = +00. Specifically, we show that for every 
L > 1 the family C contains a sub-family of L Boolean independent sets. We note that 
Boolean independence plays a related role in work of Rosenthal [12], who shows that if a 
sequence of sets {C n : n > 1} contains no pointwise convergent subsequence, then there is 
an infinite subsequence Co = {C nm : m > 1} such that each finite subfamily of Co is Boolean 
independent. 

The construction of Boolean independent sets in Theorem 1 proceeds in stages. At 
each stage a splitting set is produced by means of a weak limit, and is then incorporated 
in the construction of the splitting sets at subsequent stages. The resulting sequence of 
splitting sets is used to identify Boolean independent collections of arbitrary finite size. As 
noted by Ramon van Handel (private communication), the proof of Theorem 1 has points of 
intersection with the construction of a critical set for product measures in Theorem 11-1-1 
of Talagrand [13J, and with the notion of weakly dense sequences in Cech-complete spaces 
employed by Bourgain, Fremlin, and Talagrand [5j. Essential differences emerge from a 
number of factors, including our focus on finite approximation under a fixed (but arbitrary) 
distribution in the absence of topological structure, as well as the recursive construction of 
splitting sets that is employed in the theorem. 

1.1 Overview 

The next two sections are devoted to corollaries of Theorem[T]to families of sets and functions 
with bounded combinatorial complexity. In Section [2] we establish that VC classes of sets 
have finite bracketing numbers, and deduce similar results for VC major and VC graph 
families of functions. In Section [3] we show that VC classes satisfy uniform laws of large 
numbers for every ergodic process. The proof of Theorem Q] is presented in Section 01 

2 Bracketing of VC Classes of Sets and Functions 

Let J 7 be a family of measurable functions / : X — > K. We recall some basic definitions 
from the theory of empirical processes. A measurable function F : X — >■ [0, 00) is said to 
be an envelope for T if < F{x) for each x S X and / € T . The family T is said to 

be separable if there is a countable sub-family T§ C T such that each function f € J- is 
a pointwise limit of a sequence of functions in J-q. For each pair of measurable functions 
g, h : X — > M with g < h, the bracket [g, h] denotes the set of all measurable functions 
/ such that g < f < h pointwise on X . In particular, [g, h] is said to be an e-bracket if 
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f{h — g)d/j, < e. For e > 0, the bracketing number iVj](e, J 7 , fi) of T is the least number of 
e-brackets needed to cover T . In general, the functions defining the minimal brackets need 
not be elements of T. 

2.1 VC Classes of Sets 

Let a measure [i and family C C S be fixed. The notions of separability and bracketing may 
be applied to C if we regard its elements as indicator functions. In this case we may assume, 
without loss of generality, that the lower and upper limits of each bracket are themselves 
indicator functions. 

Corollary 1. If C is a separable VC-class, then N^(e,C,fi) is finite for every e > 0. 

Proof: By routine arguments, we may assume that C is countable. Fix e > 0. Let 
7r = {Ai, . . . , A m } be a finite measurable partition of X such that n{d{C : tt)) < e for every 
C G C, and assume without loss of generality that each cell of n has positive ^-measure. 
For each C G C, remove all points in C from Aj if n(Aj (~l C) = 0, and remove all points 
in C c from Aj if /J.(Aj n C c ) = 0. Denote the resulting set by Bj. Clearly Bj C Aj and 
[i(Aj \ Bj) = as C is countable. The definition of ensures that for each C <G C exactly 
one of the following relations holds: Bj C C, Bj C (7 C , or ^(Sj n C) • ^(Bj n C c ) > 0. 
Let B = X \ \J™ =1 Bj, and define the partition ir' = {B , Bi, . . . , B m }. Given C G C let 
Q = U{B G 7r' : B C C} and C u = U{S £ i' : B fl C / 0}. A straightforward argument 
shows that QCCC C u , and that ^(C M \ Q) = fi{d{C : tt')) = Md(C : tt)) < e. It follows 
that = {[Ci, C u ] : (7 G C} is a collection of e-brackets covering C. The cardinality of is 
at most 2 2 l 7r 'L 

2.2 VC Major Families 

Let J 7 be a family of measurable functions / : X — > M. with envelope F. For / G T and 
a G IR let Lf(a) = {x : f(x) < a} be the a-level set of /. Define 

C a = {L f (a) :feT} 

to be the family of a-level sets associated with functions in T. 

Proposition 1. Suppose that dim(C a ) < oo for every a G ML If fj, is any probability measure 
on (X,S) such that j F dfi < oo, then N^(e, J 7 , fi) < oo for every e > 0. 
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Proof: Suppose first that F is bounded, with constant envelope Af < oo. Fix e > and 
let K be an integer such that 2M/K < e. For each / € F define the approximation 

~ , 2Af , , „ 2Mj 

f(x) = M-—Y J I{x^L f {a j )) with aj = M-^-. 

3=1 

The choice of Af and fsT ensure that f(x) — e < f(x) < f(x) for each x £ X. The dimension 
of C aj is finite by assumption, and it then follows from Corollary [1] that there is a finite 
collection @j of e/2Af -brackets that covers the level sets {L/(ay) : / € J 7 }. For each / G J 
let [gj., /ij] be a bracket in Qj containing Lf(atj). With this identification, define upper and 
lower approximations of / as follows: 

/, = M-— ^^}(s)-e and /„ = AT- — 

An easy argument shows that fi<f< f u , and the family of brackets © = {[//, / u ] : / £ -T 7 } 
is finite, as |0| < Moreover, 

K 

K 



fu-fi < ^ + 



and therefore J (/ u — fi)dfi < 2e. Thus is a finite family of 2e-brackets covering F . 

Suppose now that F has an envelope F such that J Fdfj, < oo. Given e > let Af < oo be 
such that J F>M Fdix < e. For each / € F define the truncation /m(x) = (f(x) V — Af) A Af , 
and let Tm = {/a/ : / S F}. By the preceding argument, there is a finite family O of 
e-brackets covering Tm- Let [g,h] be an element of 0; without loss of generality, we may 
assume that \g\, \h\ < Af. Define 

g ' = gA (-FI{F > Af )) and h! = hV (FI(F > Af)) 

and note that g' < g < h < h! . Moreover, Jm € [g, h] implies / G [g', h'], so the finite family 
of brackets {[</, h'] : [g, h] £ 0} covers F. It is easy to see that 

h' -g' = (h- g)I(F < Af ) + 2FI(F > Af ), 

and therefore f (h! — g')dfx < J(h — g)d\i + 2 J F>M Fdfj, < 3e. 

2.3 VC Graph Families 

Let J 7 be a family of measurable functions / : X — > M with envelope F(x). The graph of 
/ € F is defined by 



G/ = {(x, s) :i£ A 1 and < s < f(x) or /(or) < s < 0} C X 



x 
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Let GiJ 7 ) = {Gf : / € J 7 } be the family of graphs of functions in T . 

Proposition 2. Suppose that dim.(Q(J r )) < oo. If [i is any probability measure on (X,S) 
such that J F d\x < oo, then Nn (e, J 7 , fj,) < oo for each e > 0. 

Proof: Suppose first that T is bounded, with constant envelope M < oo. The finiteness of 
the bracketing numbers is not affected if we replace each function / G J- by (/ + M)/2M, 
and we therefore assume that every / G T takes values in [0,1]. With this restriction, 

G f = {(x, s) : x G X and < s < f(x) < 1} C X x [0, 1]. 

Let A(-) denote Lebesgue measure on the Borel subsets B of [0, 1], and define the product 
measure v = fi <g> A on (X x [0, 1], S ® B). 

Fix e > 0. As Q{J-) has finite VC dimension, Corollary [1] ensures that G{J~) is covered 
by a finite collection of e-brackets. Without loss of generality, we may represent the 
brackets in in the form [A, B], where A, B € S <g) B and A <Z B. Let [A, B] be a bracket 
in 0. For each x € X define 

g(x) = ess-sup({s : (x,s) € A}) and h(x) = ess-inf({s : (x,s) G B c }), 

where for U C [0, 1] the essential supremum ess-sup(C/) = inf{a : fj,(UCi [0, a]) = fi(U)}, and 
ess-inf([/) is defined analogously. Routine arguments shows that g and h are measurable, 
that g < h, and that v{A\G g ) = v(B c \G'f l ) = 0. Moreover, for every function / : X — > [0, 1] 
it is easy to see that Gf G [A, B] implies Gf G [G g , Gh], which implies in turn that g < f < h. 

It follows from the arguments above that the finite family ©o of brackets [g, h] derived 
from the elements of covers T . In order to assess the size of these brackets, note that 

(G h \ G g ) x = {s : (x, s)eG h \ G g } = {s : g(x) < s < h(x)} 

and therefore by Fubini's theorem 

J (h(x) - g(x))dfi(x) = J X((G h \G g ) x )dfi(x) = v{G h \G g ) < u{B\A) < e. 

Thus every element [g, h] of ©o is an e-bracket under \i. 

The argument for an unbounded family T with an integrable envelope F is similar to 
that for VC Major families. Given e > let M < oo be such that J F>M Fdfi < e. For each 
/ G T define the truncation /m(^) = (f(x) V —M) A M, and let Tm = {fhi ■ f G J 7 }. As 
Gf M = GfC](Xx [— M, M]), it is easy to see that the dimension of Q(Fm) is no greater than 
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that of G{J-), and is therefore finite. The preceding argument shows that there is a finite 
collection of e-brackets covering Tm-, and these can be extended to 3e-brackets covering T 
following the proof of Proposition [TJ 



3 Uniform Laws of Large Numbers 

Let X = Xi, X2, ■ ■ ■ be a stationary ergodic process taking values in (Af,«S). The ergodic 
theorem ensures that for every measurable set C the sample averages n _1 Y^i=i Ic(Xi) 
converge almost surely to P(X E C). A family CCS satisfies a uniform laws of large 
numbers with respect to X if the discrepancy 



tends to zero almost surely as n tends to infinity, so that the relative frequencies of sets in 
C converge uniformly to their limiting probabilities. 

For i.i.d. processes X, Vapnik and Chervonenkis [T7j gave necessary and sufficient condi- 
tions under which A n (C : X) — > 0. For VC-classes they established exponential inequalities 



pendent of X and C. Consequently, VC classes have uniform laws of large numbers for any 
i.i.d. process. Talagrand [14J provided necessary and sufficient conditions for uniform laws 
of large numbers that strengthen those of [T7J: for non-atomic distributions, A n (C : X) -ft 
if and only if there is a set A E S with P(A) > such that with probability one C shatters 
every finite subset of {Xi : Xj E A}. 

Using the bracketing properties of VC classes established in the previous section one may 
immediately extend this result to the general ergodic case. The following theorem appears 
in Adams and Nobel pQ (under an additional Polish assumption), where there is also a 
discussion of related work on uniform laws of large numbers under dependent sampling. 

Theorem 3. If C is a separable VC-class of sets and X is a stationary ergodic process, 
then A n (C : X) — > almost surely as n tends to infinity. 

Proof: The stated convergence follows easily from Corollary Q] and standard arguments for 
the Blum DeHardt law of large numbers (c.f. \15\ [7]). 

One may establish uniform laws of large numbers for separable VC major and VC graph 
classes of functions in the general ergodic case using the bracketing results in Propositions 
[T] and [21 respectively. In pQ these results are derived directly from Theorem [3l Related 




of the form P(A n (C : X) > t) < a-n dira ^ -exp{— bt 2 }, where a, b are positive constants inde- 
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work for families of functions, under a more general, scale specific, notion of dimension can 
be found in [2]. 

4 Proof of the Main Theorem 

In the case where X is a complete separable metric space and S is the Borel subsets of X, 
one may prove Theorem [1] using arguments similar to those used in pQ to establish uniform 
laws of large numbers for VC classes under ergodic sampling. The details can be found in 
an earlier version [3] of the results presented here. Below we provide a simpler argument 
that does not require the Polish assumption. The new argument, which follows the outline 
of the proof in pQ, employs several simplifications and improvements that were suggested 
by an anonymous referee of pQ, in particular, the use of Hilbert space weak limits in the 
definition of splitting sets. 

4.1 Proof of Theorem CO 

It follows from standard results on the L p -covering numbers of VC classes (for exam- 
ple, Theorem 2.6.4 of [15]) that there exists a countable sub-family Co of C such that 
infcgco m(C'AC) = for each C € C. An elementary argument then shows that 

sup/x(<9(C : tt)) = sup n{d(C : tt)) 
cec cec 

for every finite partition tt, and we may therefore assume that C is countable. Let C = 
{Ci, C2, ■ ■ • } and let Sq = cr(C) C S be the sigma field generated by C. Suppose that the 
uniform approximation property fails to hold for C, that is, there exists a number r/ > 
such that 

sup \(d(C : tt)) > i] for every finite measurable partition tt. (1) 

Cec 

Using the inequality (JT]) we construct a sequence of "splitting sets" Si, S2, ■ ■ ■ C X from the 
sets in C in a stage-wise fashion. At the kth stage the splitting set is obtained from a 
sequential procedure that makes use of the splitting sets Si, ... , S&_i produced at previous 
stages. The splitting sets are used to identify arbitrarily large finite collections of sets in C 
having full join. The existence of these collections implies that C has infinite VC dimension 
by Lemma [Al 

First stage. Define the refining sequence of joins J\{n) = C\ V- ■ • VC„ for n > 1. It follows 
from ([T]) that for each n there is a set C\(n) € C whose boundary G\{n) = d{C\(n) : Ji(n)) 
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has measure greater than ry. Note that the sets {Gi(n) : n > 1} are measurable So- 
By standard results in functional analysis, there exists a subsequence {n m } and an (So- 
measurable function hi such that J g Ici{n m ) dfi — > J g hi dfi as m tends to infinity for every 
g € L2(X,So, /j,). (The function h\ is the weak limit of the indicator functions ^Gi(n m )-) 
It follows that < hi < 1 almost surely, and that J hi dfj, > rj. Define the splitting set 
5*i = {hi > 0} and note that //(Si) > rj. 

For simplicity, let Ji(m), Ci(m), and Gi(m) denote, respectively, the quantities Ji(n m ), 
Ci(n m ), and Gi(n m ) along the subsequence defining hi. We adopt similar notation for 
subsequences encountered at subsequent stages. 

Subsequent stages. Suppose now that we have constructed splitting sets Sj at stages 
j = 1, . . . , k — 1, and wish to construct the splitting set S& at stage k. Begin by defining the 
refining sequence of joins Jfc(n) = Si V • • ■ V Sk-i V Ci V • • • V C n for n > 1. It follows from 
(PQ) that for each n there is a set Cfc(n) E C whose boundary Gk{n) = d{Ck{n) : Jk( n )) has 
measure greater than r/. Proceeding as in Stage 1, there is a subsequence {^G fe (m)} having 
a weak limit /i^ G L^X, So, //) such that < /i& < 1 almost surely, and j hk d[i > r/. Define 
the splitting set Sfc = {hk > 0} and note that /i(Sfc) > 77. 

Construction of Full Joins. Fix an integer L > 1. As the measure of each splitting set Sk 
is at least rj, there exist positive integers fei < A% < • • • < &L+1 such that /i(n^=i St,-) > 0. 
Suppose for simplicity, and without loss of generality, that kj = j. For I = 1, . . . ,L + 1 
define 

a = n s i 

3=1 

In what follows we will make repeated use of the elementary fact that J B (hi • • • hi) dfi > 
if and only if fi(B fl Qi) > 0. 

We claim that there exist sets Z?i, . . . , Dl € C such that for each I = 1, . . . , L, 

/ (hi--- hi) dfi > for every B £ D t V • • • V D L . (2) 
Jb 

The inequalities ([2]) are established by reverse induction, beginning with the case I = L. To 
this end, note that 

< / (hi ■ ■ ■ h L +i) dfi = lim (hi ■ ■ ■ h L )I G r m) dfi, 

J m— ¥00 J tu ' 

and therefore fi(Qi H Gl+i(tti)) > for all m sufficiently large. Fix such an m and let 
D = CL+i(m). It follows from the definition of Gl+i(iti) that for some cell A £ Jj, + i(m), 

/ u(Q L nvl)>0 and fl fl) • n D c ) > 0. (3) 
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The inclusion of the sets Si, ■ ■ ■ , Sl in the definition of the joins J^ + i(n) ensures that Ql is 
a finite union of cells of JL+i(jn). The first relation in ([3]) then implies that A is necessarily 
a subset oi Ql, and it follows from the second relation that [i(Ql H I?) • ^{Ql H -D c ) > 0. 
Letting Dl = D the last inequality implies ([2]) in the case I = L. 

Suppose now that for some 1 < I < L we have identified sets Di,Di + i, . . . ,Dl such that 
([2]) holds. Then for each cell B in the join DiV ■ ■ • V Dl, 



Therefore, there exists an integer m such that fi(B n n G[(m)) > for every B € 
Di V • • • V Dl. As the join J/(m) includes the first n m elements of C, by enlarging m if 
necessary we may assume that Ji(m) includes Di, . . . ,Dl- Let D = Ci(m) and let B be 
any cell of D\ V • • • V Dl- The definition of Gi(m) implies that for some cell A E Ji(m), 



Both Qi-i and B are equal to a union of cells of the partition J\(m), so the first relation in 
([4]) implies that A C BnQi-i, and it then follows from the second relation that fi(Bf]Qi-iD 
D) and //(i? fl Qi-i fl -D c ) are positive. As these inequalities hold for each B 6 .D/ V • • -V -Dl, 
we have J B ,(hi ■ ■ ■ dfi > for every £?' S DV DjV • • -V Dl- Letting Z?j_i = D completes 
the induction. 

It follows from ([2]) that the sets D\, . . . , Dl have full join, and as L > 1 was arbitrary, 
Lemma [A] implies that C has infinite VC dimension, which completes the proof of the 
theorem. 

Remark: An inspection of the proof shows that the approximating partitions it in the 
theorem can be taken to be measurable cr(C). A simple counterexample shows that tt may 
not be chosen from the smaller family (Jn=i V C2 V ... V C n ). Let X = [0, 1] and let 
(j, be Lebesgue measure. Let 01,02,. ■■ be a sequence of positive real numbers such that 

s = Y^™=i a n < 1- Define s = and s n = Y^7=i a i for n - ancl let = [ s n-l 5 Sn)- 
Clearly, the VC-dimension of the class {Ci,C2, ■ ■ ■} equals 1, since the sets are disjoint. 
Define J n = C\ V C2 V . . . V C n . Then the set A n = [s n , 1] is a single element in J n with 
measure 1 — s ra >l — s>0. Moreover, both A n n C n+ i and A n n C^ +1 have positive 
measure. Thus, for n>l,A n C. d(C n+ i : G n ) and (j,(d(C n+ i : G n )) > 1 — s. 
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